1. Field of the Invention
This invention relates generally to the field of computer system architecture and more particularly, to an architecture that allows mapping between computing nodes and shared or non-shared I/O devices.
2. Description of the Related Art
Computing systems often contain multiple compute nodes. For example, computing systems may include multiple CPUs, one or more multi-core CPUs, CPUs that operate in multiple operating system domains, and/or multiple single-board computers configured as blades and mounted in a common chassis or drawer. In addition, computer nodes may be interfaced to multiple I/O devices. I/O devices may be any devices that allows data to be transferred or from to the compute nodes. For example, compute nodes may be coupled to one or more network interfaces such as Ethernet, storage area network interfaces such as Fibre Channel, graphics cards, USB or Firewire controllers, etc. In addition, redundant connections may also be desired to improve availability and reliability of the I/O interfaces. In modern computer systems, an interface subsystem placed between the computer nodes and the I/O devices may include a variety of chipsets connecting a host bus on the compute node side to one or more I/O buses on the other side, such as ISA, EISA, PCI, PCI-X, compact PCI, AGP, etc.
In order to make more effective use of the I/O devices in a system, the interface subsystem may be designed to permit compute nodes to share I/O devices. For instance, in a computer system that uses multiple blades to increase the available processing power, instead of placing I/O interface chipsets and I/O devices on each blade, each blade may interface to a set of shared I/O cards through a midplane that includes hardware to replace the function of the interface chipsets. The resulting architecture may provide a lower overall system cost, higher configuration flexibility, and more complete utilization of I/O devices. One skilled in the art will appreciate that a system of blades coupled to I/O devices through a midplane is but one example of an architecture in which I/O interface chipsets are separate from the compute nodes. What should be appreciated is that regardless of the type of compute nodes and I/O devices provided, some type of I/O interface permits the I/O devices to be shared. Further, the I/O interface may allow compute nodes to be designed, manufactured and sold separately from the I/O devices. Still further, the I/O interface may provide switching between compute nodes and I/O devices. Still further, the I/O interface may allow multiple compute nodes, operating independently and having one or more operating system domains, to share I/O devices as if the devices were dedicated to them.
In addition to the foregoing design considerations, efficient I/O interfaces are typically implemented in hardware or a combination of hardware and software. In the following descriptions, such I/O interfaces may be described as virtualization hardware, although it is understood that some functions of the I/O interface may comprise software and/or hardware. Virtualization hardware may typically include one or more switches to interconnect the compute nodes with the I/O devices. These switches combine together to create a virtual view of a switch fabric for each compute node. That virtual view may or may not correspond to the physical fabric layout.
One implementation of virtualization hardware uses the PCI Express protocol to interconnect compute nodes and I/O devices. In this implementation, the virtualization hardware presents a virtual view of a PCI Express system to each compute node. This virtual view contains virtual PCI Express switches for some or all of the physical PCI Express switches in the fabric. The virtual view also contains virtual I/O devices for some or all of the I/O devices in the fabric.
In a PCI Express system, a PCI switch consists of a set of virtual PCI to PCI bridges. There is one PCI to PCI bridge for each port on the PCI Express switch, either input or output. Virtual PCI express switches created by virtualization hardware also consist of PCI to PCI bridges for each port. For example, in a system with four compute nodes, each having a virtual PCI Express switch with five ports, the virtualization hardware would have to implement twenty PCI to PCI bridges. In conventional PCI Express switches, each PCI to PCI bridge is assigned a hardware data structure commonly referred to as a PCI bridge header. One way to implement a virtual PCI express switch is to assign a PCI bridge header to each port of each virtual switch. In the example of four virtual switches having five ports each, sufficient hardware for twenty bridge headers would be needed. For small numbers of virtual switches, this may be an acceptable solution. However, when larger numbers of virtual switches are created, many resources for the PCI bridge headers are required. Since the PCI Express specification defines more than 500 addressable bits in each header structure, the cost of header hardware may place a practical limit on the capabilities of virtualization hardware. In addition, if the PCI bridges are fixed for each switch port, only a single configuration is enabled. Accordingly, what is needed is a mechanism to enable bridge headers to be assigned more efficiently and in multiple configurations.